Digital twin for microphone array system

ABSTRACT

One example includes a digital twin of a microphone array. The digital twin acts as a digital copy of a physical microphone array. The digital array allows the microphone array to be analyzed, simulated and optimized. Further, the microphone array can be optimized for performing sound quality operations such as noise suppression and speech intelligibility.

FIELD OF THE INVENTION

Embodiments of the present invention generally relate to microphonearrays and sound quality operations. More particularly, at least someembodiments of the invention relate to systems, hardware, software,computer-readable media, and methods for a digital twin of a microphonearray.

BACKGROUND

A microphone array generally includes a set of microphones that arespaced apart from each other. Typically, the microphones included in themicrophone array are arranged in a particular pattern. The microphonearray works to produce an output signal or output signals based on thesounds received by the microphones in the array. Each microphone of themicrophone array can be viewed as a sensor or spatial window forreceiving an incoming signal. The output of the array is a superpositionof each element of the array and in accordance with any processingperformed on the outputs of the individual microphones or the output ofthe array.

Conventional microphone arrays operate with an acceptable level ofperformance, but there is no systematic optimization design, management,and evaluation tool that can optimize the operation and performance ofthe microphone array. Even if existing systems use test devices tomeasure an acoustic field in an environment, this is often inconvenientand does not account for changes in the environment or changes in soundthat occurs in the environment. In addition, it is often difficult tovisualize the real time performance of a microphone array. There is aneed to improve the performance of microphone arrays.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to describe the manner in which at least some of the advantagesand features of the invention may be obtained, a more particulardescription of embodiments of the invention will be rendered byreference to specific embodiments thereof which are illustrated in theappended drawings. Understanding that these drawings depict only typicalembodiments of the invention and are not therefore to be considered tobe limiting of its scope, embodiments of the invention will be describedand explained with additional specificity and detail through the use ofthe accompanying drawings, in which:

FIG. 1 discloses aspects of an environment that includes a microphoneand a digital twin of the microphone array;

FIG. 2 discloses aspects of a digital twin;

FIG. 3 discloses aspects of a digital twin operating in an environment;

FIG. 4 discloses aspects of a method for a digital twin; and

FIG. 5 discloses aspects of a computing device or a computing system.

DETAILED DESCRIPTION OF SOME EXAMPLE EMBODIMENTS

Embodiments of the present invention generally relate to microphonearrays and sound related operations. More particularly, at least someembodiments of the invention relate to systems, hardware, software,computer-readable media, and methods for sound quality operationsincluding optimizing the performance of microphone arrays.

More generally, embodiments of the invention relate to digital twins. Adigital twin is a digital copy of a physical device or system. Thefollowing discussion focuses on a digital twin of a physical microphonearray, multiple microphone arrays, or distributed microphone arrays.With the benefit of the present disclosure, any discussion of a singlemicrophone array can be applied in a similar manner to multiplemicrophone arrays and distributed microphone arrays.

The digital twin can be used to evaluate, analyze, optimize, and controlthe operation and/or performance of a physical microphone array and/orother aspects of an environment in which the microphone array operates.The optimizations made to a microphone array are often manifested inother ways. For example, a microphone array can be used to sense noisein an environment. This allows an anti-noise signal to be generated andoutput in order to cancel the noise. Thus, sound quality operations suchas optimizing noise suppression, improving speech intelligibility, andimproving or optimizing other sound quality operations can be examplesof or results of microphone array operations.

As stated in the previous example, the information collected or receivedby the microphone array can be used to generate a signal that isconfigured to cancel noise or other undesirable sounds in anenvironment. The digital twin can be used to improve the performance ofthe microphone array. Because the digital twin is a digital copy of themicrophone array, the digital twin can be augmented with functions,machine learning, and the like to improve the performance of themicrophone array and/or aspects of the environment. The digital twincan, by way of example only, analyze sound information, manipulate thesound information, perform simulations based on the sound information orbased on “what-if” sound information, optimize device settings orparameters, or the like using real time data generated by the microphonearray.

As used herein, sound information may include, but is not limited to,information, data or signals generated by a microphone array,environmental data (e.g., temperature, time), sound propagation models,environment (e.g., room) acoustic parameters, movement trajectory ofsound sources or microphone arrays, microphone array parameters such aslocation, array geometry, array shading/tapers, array steering,radiation patterns, or the like or combination thereof. Soundinformation may also include characteristics of the sound itself such assound level, sound type (e.g., speech, music, mechanical, traffic) orthe like.

A microphone array can localize sounds using, for example, beamformingmethods. Beamforming is a process that maps the position of acousticsources by steering the array transducers towards different directions.Steering is achieved by processing the signals from the individualmicrophones to extract a desired signal and to reject interferingsignals. This allows a microphone array to localize a sound source.

Advantageously, a digital twin facilitates design optimization, realtime data visualization, real time status including sound field data,speech intelligibility, latency, power consumption, warning/alerts, andthe like.

A user interface allows a user to interact with the digital twin andthus with the microphone array. The interaction of the user with thedigital twin can be implemented in the physical microphone array,simulated in the digital twin, or the like.

For example, if the microphone array is unable to effectively localize asound or a sound source in an environment, the digital twin can improvethe operation and performance of the microphone array. Using the soundinformation (or portions thereof), the digital twin may be able togenerate a recommendation to improve the performance of the microphonearray. This recommendation may be further improved with user feedback.

For example, a user may know that noise (e.g., music) is being playedfrom speakers in another room. If the microphone array (or overall soundquality system) is unable to adequately localize and suppress the noise(e.g., by generating an anti-sound signal), the visualization in theuser interface may allow the user to identify the sound source andprovide feedback. The user may indicate that the array needs to point ina different direction by clicking on a location in the user interface.This feedback allows the digital twin to change the directionality ofany beamforming methods to better cancel the music.

This can be achieved without user input as well. For an array that isunable to localize a sound, the digital twin can process the informationor data received/generated by the microphone array and then recommend achange in the array's pattern. The suggested change can be simulated bythe digital twin. The suggested change can also be visualized in a userinterface. Further, the digital twin allows the microphone array to becontrolled and adjusted remotely.

Microphone arrays may come in different configurations. Exampleconfigurations include linear microphone arrays, planar microphonearrays, and 3-D microphone arrays. Microphone arrays may be integratedwith many different devices including laptop computers, smart phones,smart speakers, interactive robots, televisions, or the like. Thedigital twin disclosed herein may be an independent application, aweb-based application, deployed with smart devices, including AR/VR(Augmented Reality/Virtual Reality) headsets and the like.

The processing performed by the digital twin can be performed locally(e.g., at the microphone array), at a local device connected with themicrophone array, at an edge server, in the cloud or the like. With adigital twin, a three-dimensional model of an environment or structurecan be visualized and real time data and/or status for sound field datacan be provided. Further, speech intelligibility, latency, powerconsumption, warnings, alerts, or the like can be provided or performed.

The digital twin may be configured to simulate a sound field based oninputs and allows the design of or configuration of microphone arrays tobe optimized. The digital twin may include machine learning models andmay be capable of performing data analytics. For example, the machinelearning models perform analysis using or based on historical data,real-time data, and existing databases, predict future trends, and makesuggestions for noise suppression or speech enhancement. The digitaltwin enables remote monitoring and remote control. This may allow a userto change or adjust parameters or settings for a real or physicalmicrophone array remote by controlling the digital twin.

FIG. 1 illustrates an example of an environment that includes amicrophone array and a digital twin of the microphone array. Theenvironment 100 in FIG. 1 may be a room such as an office or aconference room, a building, a room of a house, or other area orlocation. In this example, a user 104 may be using a device 102 (e.g.,laptop, tablet, smart phone or the like) for activities such as aconference call, an online meeting, streaming video and/or audio, or thelike. A microphone array 106 may be integrated into the device 102. Thearray 106, may include one or more individual microphones.

In addition, microphone arrays 108 and/or 110 may be deployed in theenvironment 100. Each of the arrays 108 and 110 may include one or moreindividual microphones. Any number of microphone arrays may be deployedin the environment 100.

In this example, the arrays 108 and 110 may each have a wired/wirelessconnection with the device 102 and the device 102 may have a wiredand/or wireless connection to the cloud 120. In this example, the device102 is connected to a server 122 (e.g., an edge server, a datacenterserver, a cluster or the like). Information or data from the microphonearrays may be delivered to the device directly, to the cloud via thedevice, and/or to the cloud directly.

The device 102 may also include or have access to a digital twin 130,which includes at least a digital copy of the array 106, the array 108,and/or the array 110. The digital twin 130 may include a sound engine132 that is configured to perform sound quality operations using soundinformation. The sound engine 132 may include a machine learning model.

The computing resources needed by the digital twin 130 to perform thesound quality operations may be implemented at the device 102 and/or atthe server 122 or elsewhere in the cloud or in an edge server or othersuitable location or device. In some examples, the microphone arrays mayalso have some processing resources. In some examples, the digital twin130 may be implemented at the device 102 only, in the cloud 120 only, orin both the device 102 and the cloud 120. The server 122, which may bean edge server, may perform some of the operations of or functions ofthe digital twin 130 or of the sound engine 132.

FIG. 1 illustrates environments 150 and 160, which may accommodate,respectively, users 152 and 154. The environment 150 may include anarray 154 and the environment 160 may include an array 164. Theenvironments 150 and 160 may also be associated with digital twins.Alternatively, the digital twin 130 may be configured to include digitalcopies of multiple microphone arrays in different environments. Thisallows the digital twin 130 to operate with distributed microphonearrays. Alternatively, multiple digital twins may operate as afederation.

FIG. 1 illustrates that the user 104 may be involved in communications,such as an online conference call, with users in other environments,such as the users 152 and 162. The array 106 may operates, by way ofexample, to ensure that speech of the users 152 and 154 can be heard bythe user 104. This is achieved, in part, by generating a cancellationsignal based on the sounds sensed by the array 106 (and/or the arrays108 and 110. For example, the microphone array 106 may be used to cancelnoise 112 in the environment 100. This allows the user 104 to hearspeech of the users 152 and 162 that is output by the device 102.Further, the arrays 154 and 164 can be used to enhance theintelligibility of the speech of the users 152 and 162 that is deliveredto the user 104 via the device 102. The digital twin 130 can operate toimprove the performance of the array 106 and/or the arrays 154 and 164.Thus, the digital twin 130 can improve cancellation of the noise 112,prevent the noise 112 from impacting the ability of the user 104 to hearspeech. The speech intelligibility of the user 152, as heard by otherusers, is improved by performing complementary sound operations inmultiple environments.

The digital twin 130 is configured to present the user 104 with avisualization of the sound in the environment 100. The digital twin 130may include or have access to models including room models, arraymodels, sensor models, acoustic models, propagation models, or the like.The digital twin 130 can be built or operated using historical data,real time performance data, environmental data, user data, userfeedback, and the like.

FIG. 2 illustrates an example of a digital twin implemented in anenvironment. The digital twin 270 may be configured to receive inputs202 (examples of sound information) such as noise source data 204,background noise 206, movement trajectory 208, array parameters 210,room acoustic parameters 212, sound propagation model 214, and the likeor combination thereof. Some of these inputs 202 may be received via amicrophone array.

The noise source data 204 may include information about different typesof noise such as transportation noise, construction noise, domesticnoise, or the like. The background noise 206 may include the sounds ornoise captured by the physical microphone array. The movement trajectory208 may include information related to movement of sound sources in theenvironment. The array parameters 210 may include locations of thearrays, geometry of the arrays, array shading and tapers, arraysteering, radiation patterns, or the like. The room acoustic parameters212 may include sound strength, reverberance, clarity, and the like. Thesound propagation model 214 may include information related to how soundpropagates in an environment.

The inputs 202 allow the digital twin 270 to digitally mimic theoperation of the physical microphone array. With these inputs 202, thedigital twin 270 can perform various functions 250 including, but notlimited to, data retrieval 242, optimization 248, simulation 244,sensing 250, performance prediction 246, and remote monitoring 252.

Data retrieval 242 may include real-time data retrieval. The dataretrieved may include sound information such as location, loudness, orthe like. Optimization 248 may include making adjustments to improve theperformance of the physical microphone array such that the soundenvironment is improved (e.g., better noise cancellation, better echocancellation, improved speech intelligibility).

The simulation 244 allows the impact of changes or adjustments to themicrophone array to be previewed. For example, the simulation 244 mayallow the digital twin 270 to determine how a change in the arrayparameters (e.g., array pattern change) may impact the performance ofthe array or more generally how the changes impact. In other words, willa change in the array pattern improve or worsen noise cancellation orsound localization. More generally, the array is adjusted to ensure thatthe user is able to hear the desired sound clearly while reducing theimpact of undesired sounds or signals.

Sensing 250 allows the digital twin 270 to identify sound sources andlocations, source new sound sources, or the like. The performanceprediction 246 indicates how the array will perform over time. Forexample, a moving sound source may not be adequately cancelled andperformance may be expected to worsen absent adjustment. The remotemonitoring 252 allows a remote entity to potentially control themicrophone array. For example, an employer may adjust an employee'smicrophone array to improve the work environment. Similarly, onlinelearning could be improved so that remote students can better understandthe teacher or comments from other students.

The digital twin 270 may also generate outputs 220. The outputs 220 mayinclude a sound map 222, a real-time data visualization 224, optimizedarray patterns/parameters 226, speech intelligibility 228, SNR/RIR 230,and sound/audio quality 232. The outputs 220 may include an acousticfield or sound map that identifies locations and levels of sound ornoise sources and room impulse responses.

The outputs 220 may include information related to single microphonearrays and/or to distributed microphone arrays. This information mayinclude optimizing the microphone array directivity index, spatialstructures, filter weights, and the like. Spatial resolution, speechindelibility, power consumption, directivity index, and the like areexamples of outputs 220. The digital twin 270 may also provide objectivemetrics such as PESQ (perceptual evaluation of speech quality), STOI(short-time objective intelligibility) and frequency weighted SNR(Signal-noise ratio).

Optimizations can be made manually or automatically. Manual optimizationmay be based on simulations. A user can input different parameters intothe digital twin 270, an example of the digital twin 130, to runsimulations and output a simulated performance index such as noiselevel, speech intelligibility and the like. Based on the simulatedresults, a user can manually select and implement the best options fornoise suppression and speech enhancement. These options or changes mayrelate to microphone sensitivity, array pattern, beamforming parameters,or the like.

Automatic optimization may be based on machine learning. The digitaltwin 270 may include a machine learning model (e.g., the sound engine)that can generate insights, optimize noise suppression and speechenhancement. The digital twin can provide noise cancellation decisionsand monitor processes remotely. More specifically in one example, thedigital twin may generate recommended array changes to improve noisecancellation or for other optimizations. As input is continuallyreceived into the digital twin, the digital twin 270 can continuallymake adjustments to the microphone array. The digital twin 270 may alsoperform simulation or perform other sound quality operations.

The digital twin 270 is also associated with a user interface 260. Theuser interface 260 may be web based, 3D model based, AR glass, or thelike. The outputs 220 can be presented visually in the user interface260. The user interface 260 may present relative locations of soundsources, human speech, noise, or the like visually. Sound informationmay also be provided in the user interface 260 for each of the soundsources. The sound information or outputs may be overlaid onto the soundsources in the display.

The user can interact with the sound information in the user interface260. For example, selecting a sound source may allow more informationabout the selected sound source to be viewed. A user can simulate ormake actual changes to the displayed information, or the like. A usermay provide feedback that a particular sound source is not sufficientlycancelled or is not properly located, or the like.

The digital twin 270 allows the status and performance of the microphonearray to be visualized in real time. The results of simulations can beviewed. Assumptions about noise suppression and speech enhancement canbe tested in the digital twin 270.

FIG. 3 illustrates an example of a digital twin operating in anenvironment. FIG. 3 illustrates a device 320 side and cloud or edge 322side. The device side 320 relates to the user 306 and the user'senvironment. The edge 322 side refers to the edge or the cloud, in whichthe digital twin 312 may be at least partially implemented. This arrays308 operate in the user's environment while allowing processing,including compute intensive processing, to be performed in the edge 322,which has more powerful computing resources.

In this example, microphone arrays 308 are present in an environment ofthe user 306. The user 306 may also be associated with a user interfacesuch as AR/VR devices 310. The microphone arrays 308 represent themicrophone array associated with the user 306 as well as microphonearrays that may be associated with remote users. Inputs to themicrophone arrays 308 may include background noise 302 and speech fromremote participants 304. More specifically, the microphone array 308associated with the user 306 may sense sound in the user's environment.In addition, speech from remote participants 304 that is output byspeakers associated with the user 306 may also be sensed or picked up bythe microphone array 308. The microphone arrays of other users mayoperate similarly.

The output of the microphone arrays 308 is input to a digital twin 312.The digital twin 312 may have access to Al based analytics 316 (e.g., amachine learning model such as a sound engine) that is trained onhistory or expert 314 data. By extracting features from the output ofthe microphone arrays 308, the analytics 316 may provide insights orother output to a decision engine 318 that may make a decision orrecommendation. For example, the analytics 316 may recommend a change inthe pattern of the microphone arrays 308 (e.g., turn specificmicrophones on/off). This decision can be provided to the digital twin312 and simulated if desired. This decision can also be implemented atthe microphone arrays 308.

The digital twin 312 may also provide other outputs 324 to userinterface devices such as AR/VR devices 310. This may include real timesound visualization, 3D models of microphone array patterns, warnings,alerts, predictions, and suggestions. The sound information included inthe output 310 may be overlaid onto sound sources in the devices 310 orpresented to the user. The user can then interact with these outputs.For example, in setting up a classroom environment with multiple relateddevices, an administrator could model multiple scenarios with differentnoise levels, different noise locations, and different array placementsto determine the result. This type of simulation and testing can allow aconfiguration to be implemented that is most conducive to learning andthat best reduces or suppresses noise while enhancing desired signalssuch as desired speech.

The digital twin 312 may include or have access to different modelsincluding reduced order models for microphone arrays that can be used asa visualization and interaction interface, a 3D model for environment orroom layout, and 3D models for other sensors if present. The data usedto construct the digital twin 312 may include historical data and realtime performance data, environmental data, user data, User feedback,noise source data, movement trajectory, or the like.

Simulations can be used to simulate the acoustic environment and themicrophone array performance. Machine learning or artificialintelligence can be used to implement noise suppression, speechenhancement, anomaly prediction, configuration optimization or the like.

Using AR/VR devices 310 with the digital twin 312 allows data to bevisualized in a 3D environment. This type of user interface allows theuser to interact with the model through a virtual environment. The usercan view the 3D model and the data in real-time from multipleperspectives. Data that can be visualized includes microphone arraysystem model and architecture, data that are interpreted andcontextualized, and simulation outputs. Analyzed data, warnings, alerts,operation suggestions can also be visualized. These different categoriesof virtual information can be automatically popped up in the device. Auser can use gestures or other controls to pull out information thatinterests the user. User can interact with them, get insights, makedecisions, test assumptions, run simulations, and/or take other actions.The visualization could also be independent application or could be webbased.

Pairing the virtual digital twin with a physical microphone array allowsanalysis of data and monitoring of systems to head off problems beforethey even occur, prevent downtime, develop new opportunities, and evenplan for the future by using simulations.

Using data from multiple sources, a digital twin continuously learns andupdates itself to represent the current working condition of themicrophone array system, make optimization suggestions, make predictionsabout future status, or the like. The machine learning based analyticsresults in suggestions for noise suppression and speech enhancement, andmakes predictions for future trends, such as: suggestions onoptimization of microphone array patterns or types (e.g. periodical,nonperiodical, and logarithmic microphone line arrays, etc.),recommendations for current & future adjustments for settings,predictions on future room acoustics performance, or the like.

The data sources for the machine learning model, or the digital twin mayinclude: physical parameters of microphone array, sensing data,historical and real time sound/noise source parameters, historical andreal time microphone array performance data, movement trajectory ofsources and microphone arrays in room, expert models or knowledgedatabase, user feedback on settings, remote monitoring andcollaboration, or the like or combination thereof.

With a digital twin, the user does not need to access the physicalsystem to check the status or performance. Status and performance can bechecked in the digital twin, which allows for remote monitoring.

When an issue occurs, the digital twin provides remote collaboration.The visualization can be shared for example. This allows the user and/ora remote user to try and resolve the issue or to simply work together.

The digital twin provides visibility into the performance of themicrophone array. The digital twin can predict future state of themicrophone array system using simulation and modeling. The digital twinallows a user to interact with the digital twin to perform a what-ifanalysis. Optimal solutions and setups can be and developed. The digitaltwin also allows the microphone array to be cope and adapt to changingenvironments. The real time modeling performed by the digital twinallows beamforming to be conducted on-the-fly to dynamically focus themicrophones and optimize noise cancelling and perform other functions.The real time modeling could be used in advance to predict the correctstarting settings when a new environment is set up and/or there areprojected changes to an existing environment.

The digital twin includes machine learning based optimizations, whichmay be automatic, and manual optimizations. The digital twin cangenerate insights for users to optimize noise suppression and speechenhancement.

The digital twin also enable remote monitoring. In one example, thisallows products to be diagnosed or operated remotely. This may lowercosts and improve customer satisfaction. AR/VR allows users tomanipulate the microphone array systems. However, users could opt-out ofremote monitoring.

Generally, the digital twin may ingest large amounts of data and AR andVR provide immersive experiences to visualize these insights. Further,visualization can be performed at multiple layers. For example, an enduser view may provide visualization into the microphone array of the enduser. A view may be provided into all systems in a network (anadministrator view).

Digital twins involve ingesting large volumes of data to arrive atactionable insights. AR and VR offer immersive experiences to visualizesuch insights. A stakeholder can interact with a wearable or a hand-heldAR/VR device to consume insights and queries in a context-aware manner.Integrating AR/VR with Digital twin can help users quickly andeffectively view information in context from the digital twin.

The digital twin allows remote interaction with microphone arrays atvarious levels, such as the end user's system (end user view, with justone system), all systems within a network (admin view) or the like. Witha user's permission, multiple arrays can be viewed and adjustedsimultaneously.

FIG. 4 discloses aspects of a method for a digital twin. Initially,inputs are provided to or received 402 by a digital twin in the method400. The inputs may be provided by a microphone array. For example, themicrophone may sense sound from sound sources such as background noise,speech from other users (played from a user's device) or the like. Theinputs to the digital twin may also include other factors that do notcome from the microphone array. The inputs may also include arrayparameters, steering parameters, or the like.

Next, the digital twin performs 404 sound quality operations on theinputs. The sound quality operations may include, but are not limitedto, sound localization operations, noise cancellation operations, whichmay include generating a cancellation signal, simulation operations,speech enhancement operations, array pattern operations, remotemonitoring operations, or the like.

After performing the sound quality operations, which may be performedrepeatedly or continually, outputs are generated 406. The outputs may beimplemented as well. The outputs may include adjustments to themicrophone array, a noise cancellation signal, speech improvement, soundlocations, sound types, and the like.

In some examples, these operations are performed using machine learning.A machine learning model, trained using historical and/or expert data orthe like.

Embodiments of the invention, such as the examples disclosed herein, maybe beneficial in a variety of respects. For example, and as will beapparent from the present disclosure, one or more embodiments of theinvention may provide one or more advantageous and unexpected effects,in any combination, some examples of which are set forth below. Itshould be noted that such effects are neither intended, nor should beconstrued, to limit the scope of the claimed invention in any way. Itshould further be noted that nothing herein should be construed asconstituting an essential or indispensable element of any invention orembodiment. Rather, various aspects of the disclosed embodiments may becombined in a variety of ways so as to define yet further embodiments.Such further embodiments are considered as being within the scope ofthis disclosure. As well, none of the embodiments embraced within thescope of this disclosure should be construed as resolving, or beinglimited to the resolution of, any particular problem(s). Nor should anysuch embodiments be construed to implement, or be limited toimplementation of, any particular technical effect(s) or solution(s).Finally, it is not required that any embodiment implement any of theadvantageous and unexpected effects disclosed herein.

The following is a discussion of aspects of example operatingenvironments for various embodiments of the invention. This discussionis not intended to limit the scope of the invention, or theapplicability of the embodiments, in any way.

In general, embodiments of the invention may be implemented inconnection with systems, software, and components, that individuallyand/or collectively implement, and/or cause the implementation of, dataprotection operations which may include, but are not limited to, datareplication operations, IO replication operations, dataread/write/delete operations, data deduplication operations, data backupoperations, data restore operations, data cloning operations, dataarchiving operations, and disaster recovery operations. More generally,the scope of the invention embraces any operating environment in whichthe disclosed concepts may be useful.

Example cloud computing environments, which may or may not be public,include storage environments that may provide data protectionfunctionality for one or more clients. Another example of a cloudcomputing environment is one in which processing, data protection, andother, services may be performed on behalf of one or more clients. Someexample cloud computing environments in connection with whichembodiments of the invention may be employed include, but are notlimited to, Microsoft Azure, Amazon AWS, Dell EMC Cloud StorageServices, and Google Cloud. More generally however, the scope of theinvention is not limited to employment of any particular type orimplementation of cloud computing environment.

In addition to the cloud environment, the operating environment may alsoinclude one or more clients that are capable of collecting, modifying,and creating, data. As such, a particular client may employ, orotherwise be associated with, one or more instances of each of one ormore applications that perform such operations with respect to data.Such clients may comprise physical machines, or virtual machines (VM),or containers.

Particularly, devices in the operating environment may take the form ofsoftware, physical machines, containers, or VMs, or any combination ofthese, though no particular device implementation or configuration isrequired for any embodiment. Similarly, data protection systemcomponents such as databases, storage servers, storage volumes (LUNs),storage disks, replication services, backup servers, restore servers,backup clients, and restore clients, for example, may likewise take theform of software, physical machines or virtual machines (VM), though noparticular component implementation is required for any embodiment.

As used herein, the term ‘data’ is intended to be broad in scope. Thus,that term embraces, by way of example and not limitation, data segmentssuch as may be produced by data stream segmentation processes, datachunks, data blocks, atomic data, emails, objects of any type, files ofany type including media files, word processing files, spreadsheetfiles, and database files, as well as contacts, directories,sub-directories, volumes, and any group of one or more of the foregoing.

Example embodiments of the invention are applicable to any systemcapable of storing and handling various types of objects, in analog,digital, or other form.

It is noted that any of the disclosed processes, operations, methods,and/or any portion of any of these, may be performed in response to, asa result of, and/or, based upon, the performance of any precedingprocess(es), methods, and/or, operations. Correspondingly, performanceof one or more processes, for example, may be a predicate or trigger tosubsequent performance of one or more additional processes, operations,and/or methods. Thus, for example, the various processes that may makeup a method may be linked together or otherwise associated with eachother by way of relations such as the examples just noted. Finally, andwhile it is not required, the individual processes that make up thevarious example methods disclosed herein are, in some embodiments,performed in the specific sequence recited in those examples. In otherembodiments, the individual processes that make up a disclosed methodmay be performed in a sequence other than the specific sequence recited.

Following are some further example embodiments of the invention. Theseare presented only by way of example and are not intended to limit thescope of the invention in any way.

Embodiment 1. A method, comprising: receiving inputs into a digitaltwin, the inputs including sound information from a microphone array,performing sound quality operations based on the inputs by the digitaltwin, wherein the digital twin includes a digital copy of the microphonearray, generating outputs of the sound quality operations.

Embodiment 2. The method of embodiment 1, further comprising presentinga visual representation of the outputs in a user interface of a device.

Embodiment 3. The method of embodiment 1 and/or 2, wherein the devicecomprises an augmented reality or virtual reality device.

Embodiment 4. The method of embodiment 1, 2, and/or 3, wherein theinputs include background noise, speech from remote participants, speechfrom other users in the environment, environment acoustics parameters,movement trajectory, a sound wave propagation model, and arrayparameters.

Embodiment 5. The method of embodiment 1, 2, 3, and/or 4, wherein thearray parameters include a location of the microphone array, an arraygeometry, array steering, and radiation patterns.

Embodiment 6. The method of embodiment 1, 2, 3, 4, and/or 5, furthercomprising generating an sound map that identifies locations and levelsof noise sources and room impulse responses.

Embodiment 7. The method of embodiment 1, 2, 3, 4, 5, and/or 6, furthercomprising generating, for the microphone array, optimized parametersincluding array pattern directivity index, spatial structure, filterweights, spatial resolution, speech indelibility, power consumption,perceptual evaluation of speech quality, short-time objectiveintelligibility, and frequency-weighted signal to noise ratio.

Embodiment 8. The method of embodiment 1, 2, 3, 4, 5, 6, and/or 7,further comprising receiving feedback from a user and performing thesound operations using the user feedback.

Embodiment 9. The method of embodiment 1, 2, 3, 4, 5, 6, 7, and/or 8,further comprising performing a simulation by the digital twin.

Embodiment 10. A method for performing any of the operations, methods,or processes, or any portion of any of these, or any combinationthereof, disclosed herein.

Embodiment 11. A non-transitory storage medium having stored thereininstructions that are executable by one or more hardware processors toperform operations comprising the operations of any one or more ofembodiments 1 through 10.

The embodiments disclosed herein may include the use of a specialpurpose or general-purpose computer including various computer hardwareor software modules, as discussed in greater detail below. A computermay include a processor and computer storage media carrying instructionsthat, when executed by the processor and/or caused to be executed by theprocessor, perform any one or more of the methods disclosed herein, orany part(s) of any method disclosed.

As indicated above, embodiments within the scope of the presentinvention also include computer storage media, which are physical mediafor carrying or having computer-executable instructions or datastructures stored thereon. Such computer storage media may be anyavailable physical media that may be accessed by a general purpose orspecial purpose computer.

By way of example, and not limitation, such computer storage media maycomprise hardware storage such as solid state disk/device (SSD), RAM,ROM, EEPROM, CD-ROM, flash memory, phase-change memory (“PCM”), or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other hardware storage devices which may be used tostore program code in the form of computer-executable instructions ordata structures, which may be accessed and executed by a general-purposeor special-purpose computer system to implement the disclosedfunctionality of the invention. Combinations of the above should also beincluded within the scope of computer storage media. Such media are alsoexamples of non-transitory storage media, and non-transitory storagemedia also embraces cloud-based storage systems and structures, althoughthe scope of the invention is not limited to these examples ofnon-transitory storage media.

Computer-executable instructions comprise, for example, instruction anddata which, when executed, cause a general purpose computer, specialpurpose computer, or special purpose processing device to perform acertain function or group of functions. As such, some embodiments of theinvention may be downloadable to one or more systems or devices, forexample, from a website, mesh topology, or other source. As well, thescope of the invention embraces any hardware system or device thatcomprises an instance of an application that comprises the disclosedexecutable instructions.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts disclosed herein are disclosed asexample forms of implementing the claims.

As used herein, the term ‘module’ or ‘component’ may refer to softwareobjects or routines that execute on the computing system. The differentcomponents, modules, engines, and services described herein may beimplemented as objects or processes that execute on the computingsystem, for example, as separate threads. While the system and methodsdescribed herein may be implemented in software, implementations inhardware or a combination of software and hardware are also possible andcontemplated. In the present disclosure, a ‘computing entity’ may be anycomputing system as previously defined herein, or any module orcombination of modules running on a computing system.

In at least some instances, a hardware processor is provided that isoperable to carry out executable instructions for performing a method orprocess, such as the methods and processes disclosed herein. Thehardware processor may or may not comprise an element of other hardware,such as the computing devices and systems disclosed herein.

In terms of computing environments, embodiments of the invention may beperformed in client-server environments, whether network or localenvironments, or in any other suitable environment. Suitable operatingenvironments for at least some embodiments of the invention includecloud computing environments where one or more of a client, server, orother machine may reside and operate in a cloud environment.

With reference briefly now to FIG. 5 , any one or more of the entitiesdisclosed, or implied, by the Figures and/or elsewhere herein, may takethe form of, or include, or be implemented on, or hosted by, a physicalcomputing device, one example of which is denoted at 500. As well, whereany of the aforementioned elements comprise or consist of a virtualmachine (VM), that VM may constitute a virtualization of any combinationof the physical components disclosed in FIG. 5 .

In the example of FIG. 5 , the physical computing device 500 includes amemory 502 which may include one, some, or all, of random access memory(RAM), non-volatile memory (NVM) 504 such as NVRAM for example,read-only memory (ROM), and persistent memory, one or more hardwareprocessors 506, non-transitory storage media 508, UI device 510, anddata storage 512. One or more of the memory components 502 of thephysical computing device 500 may take the form of solid state device(SSD) storage. As well, one or more applications 514 may be providedthat comprise instructions executable by one or more hardware processors506 to perform any of the operations, or portions thereof, disclosedherein.

Such executable instructions may take various forms including, forexample, instructions executable to perform any method or portionthereof disclosed herein, and/or executable by/at any of a storage site,whether on-premises at an enterprise, or a cloud computing site, client,datacenter, data protection site including a cloud storage site, orbackup server, to perform any of the functions disclosed herein. Aswell, such instructions may be executable to perform any of the otheroperations and methods, and any portions thereof, disclosed herein.

The present invention may be embodied in other specific forms withoutdeparting from its spirit or essential characteristics. The describedembodiments are to be considered in all respects only as illustrativeand not restrictive. The scope of the invention is, therefore, indicatedby the appended claims rather than by the foregoing description. Allchanges which come within the meaning and range of equivalency of theclaims are to be embraced within their scope.

What is claimed is:
 1. A method, comprising: receiving inputs into adigital twin, the inputs including sound information from a microphonearray deployed in an environment; performing sound quality operationsbased on the inputs by the digital twin, wherein the digital twinincludes a digital copy of the microphone array; generating outputs ofthe sound quality operations.
 2. The method of claim 1, furthercomprising presenting a visual representation of the outputs in a userinterface of a device.
 3. The method of claim 2, wherein the devicecomprises an augmented reality or virtual reality device.
 4. The methodof claim 1, wherein the inputs include background noise, speech fromremote participants, speech from other users in the environment,environment acoustics parameters, movement trajectory, a sound wavepropagation model, and array parameters.
 5. The method of claim 4,wherein the array parameters include a location of the microphone array,an array geometry, array steering, and radiation patterns.
 6. The methodof claim 1, further comprising generating an sound map that identifieslocations and levels of noise sources and room impulse responses.
 7. Themethod of claim 6, further comprising generating, for the microphonearray, optimized parameters including array pattern directivity index,spatial structure, filter weights, spatial resolution, speechindelibility, power consumption, perceptual evaluation of speechquality, short-time objective intelligibility, and frequency-weightedsignal to noise ratio.
 8. The method of claim 1, further comprisingreceiving feedback from a user and performing the sound operations usingthe user feedback.
 9. The method of claim 1, further comprisingperforming a simulation by the digital twin.
 10. A non-transitorystorage medium having stored therein instructions that are executable byone or more hardware processors to perform operations comprising:receiving inputs into a digital twin, the inputs including soundinformation from a microphone array; performing sound quality operationsbased on the inputs by the digital twin, wherein the digital twinincludes a digital copy of the microphone array; generating outputs ofthe sound quality operations.
 11. The non-transitory storage medium ofclaim 10, further comprising presenting a visual representation of theoutputs in a user interface of a device.
 12. The non-transitory storagemedium of claim 11, wherein the device comprises an augmented reality orvirtual reality device.
 13. The non-transitory storage medium of claim10, wherein the inputs include background noise, speech from remoteparticipants, room acoustics parameters, movement trajectory, a soundwave propagation model, and array parameters.
 14. The non-transitorystorage medium of claim 13, wherein the array parameters include alocation of the microphone array, an array geometry, array steering, andradiation patterns.
 15. The non-transitory storage medium of claim 10,further comprising generating an sound map that identifies locations andlevels of noise sources and room impulse responses.
 16. Thenon-transitory storage medium of claim 16, further comprisinggenerating, for the microphone array, optimized parameters includingarray pattern directivity index, spatial structure, filter weights,spatial resolution, speech indelibility, power consumption, perceptualevaluation of speech quality, short-time objective intelligibility, andfrequency-weighted signal to noise ratio.
 17. The non-transitory storagemedium of claim 10, further comprising receiving feedback from a userand performing the sound operations using the user feedback.
 18. Thenon-transitory storage medium of claim 10, further comprising performinga simulation by the digital twin.
 19. A non-transitory storage mediumhaving stored therein instructions that are executable by one or morehardware processors to implement a digital twin of a physical device,the digital twin comprising: an input configured to receive noise sourcedata, background noise, movement trajectory, array parameters, roomacoustic parameters, and a sound propagation model; functions includingdata retrieval functions, optimization functions, simulation functions,sensing functions, performance prediction functions, and remotemonitoring functions; an output configured to generate outputs includinga sound map, real-time data visualization, optimized array patterns andparameters; speech intelligibility, signal to noise ratio, and soundquality.
 20. The non-transitory storage medium of claim 19, the digitaltwin further comprising a user interface configured to visualize atleast some of the outputs.